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ABSTRACT 



This guide is designed to help child care partnerships 
understand three different ways of using results-based decision making (to 
manage their own work, to manage the work of others, and to contribute to the 
work of others), and how they can use this process most effectively to 
improve the lives of children and families. The guide is intended to help 
partnerships recognize and manage the political issues around results-based 
decision making, and understand enough about the technical considerations to 
be able to ask good questions and seek more information. The guide is 
organized into the following sections: (1) introduction; (2) benefits of and 
cautions about using results-based decision making; (3) the language of 
results- based decision making; (4) the creation of "logic models," which are 
frameworks for linking the partnership's desired results with activities to 
effect change; (5) guidelines for using results-based decision making to 
improve performance; (6) a phased approach for using results-based decision 
making to improve the lives of children and families; and (7) conclusion. The 
appendices include resources for more detailed information on specific topics 
(such as instruments to measure collaboration, quality of child care, child 
development, family functioning, and parenting skills) . Contains 22 
references. (EV) 
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Introduction 



Across the country, public and private sector partners are joining 
forces to improve child care as part of efforts to improve the lives 
of children and families in their community. These partnerships 
take many forms, and have diverse plans for helping young 
children — strategies include increasing public awareness of the 



importance of high quality care, spending money 
on more or better child care services, and advocat- 
ing for changes in the way government, employers 
and others support high quality care. Each part- 
nership may have a different action plan and a 
different definition of success. But what they all 
have in common is a desire to see results, and a 
need to know that their work is on the right track 
to achieve those results. 



The next section defines many of the key 
terms in detail, but a brief synopsis here will 
help. A “result” (or outcome) is a “broad 
condition of well-being for families, children 
or communities,” such as children ready for 
school. An “indicator” is a measure which 
helps quantify^community-wide progress 
towards the result, such as children reaching 
developmental milestones. “Performance 
measures” track one person’s, program’s or 
organization’s contribution to that progress. 
These and other terms are also defined in 
the glossary. 



To do this, a partnership needs to use a new 
way of measuring and achieving success, called 
“results-based decisionmaking.” Using results- 
based decisionmaking means that a partnership 



decides to organize its work around results for 
children and families; chooses the results, indicators 
and performance measures on which it will focus; 
implements the actions it thinks are necessary; 
measures its progress; and uses this information to 
constandy improve its work. It means the partner- 
ship measures progress based not only on activity, 
but also on whether it is making a positive diflFer- 
ence in the lives of children and families. Changing 
this definition of success is fundamental to results- 
based decisionmaking. 

This change might seem like a long-overdue shift 
that everyone would embrace with open arms. Yet, 
some partnerships are less than enthusiastic — 
wary even — about results-based decisionmaking. 
Even though using results in this way can help a 
partnership accomplish its aims, it sounds compli- 
cated, challenging and time-consuming. 

But there are many reasons to forge on-there are 
great rewards in knowing where the partnership is 
headed and whether it is making progress. Using 
results has the potential to make the partnership 
not only more knowledgeable about where it is 
headed, but more likely to get there. And this guide 
can help the partnership avoid common pitfalls 
while using results-based decisionmaking to its 
fullest advantage. 
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How CAN PUBLIC-PRIVATE 
PARTNERSHIPS USE RESULTS- 
BASED DECISIONMAKING? 

In general, public-private child care partnerships 
form when the public sector — federal, state, local 
and/or tribal officials or agencies — joins with the 
private sector — families, employers, philanthropies, 
media, civic groups, and/or service providers — to 
attain a shared goal of improving child care in their 




state or community. The parmership may be formal- 
ly created by an official body, such as a governor, 
legislature or mayor; it may form when a few com- 
munity leaders (such as business leaders) decide to 
create such a body and then invite others to join; 
or it may form as a group of peers who see a joint 
mission. Although each parmership is unique, 
they typically share some common characteristics: 

■ They bring together public- and private- 
sector partners 

■ Partners work together towards shared goals 
or objectives 

■ Each partner contributes time, money, expertise 
and/or other resources 

■ Decisionmaking and management responsi- 
bilities are shared among the partners 



These diverse origins and structures mean that 
partnerships may be accountable for improving 
results in many different ways to various groups 
of people — they may have very little direct 
accountability or more formal responsibility for 
specific goals. They may therefore use results- 
based decisionmaking in a variety of ways. 

However, it is important that any accountability 
relationship be a two-way partnership, rather than 
an absolute transfer of responsibility. Improving 
the lives of children and families is difficult work, 
and no one has guaranteed answers. Creating a fair 
and effective svstem means sharing responsibility 
between those who manage changes and those who 
work to produce them (Iowa leaders use the terms 
“results brokers” and “results producers”). It means 
pooling knowledge and making joint decisions 
about what works for children and families. It does 
not mean leading with the threat to criticize or 
reduce funds but rather with the joint mission of 
improving services and thereby results for children 
and families. 

It mav be helpful to think about three general 
ways partnerships can use results-based decision- 
making to help them move towards their goals. 



Creating a fair and effective system 
sharing responsibility between “results birolcei^ 
and “results producers.” ^ 



1. To MANAGE THEIR OWN WORK — 

AS A PARTNERSHIP — TO IMPROVE THE 
LIVES OF CHILDREN AND FAMILIES. 

Child care partnerships are created, by themselves 
or others, to improve some aspect of childrens 
lives. The first type of joint accountability, then, is 
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to themselves or an outside body for their work. 
This outside body is likely to be the entity that 
created or funded the partnership, such as the 
Governor, legislature or a foundation. If there is 
no formal, outside body to which the partnership 
is accountable, it may hold itself accountable for 
its work — or expect that the public would hold it 
accountable for what it hopes to accomplish. 

Partnerships may take on responsibility for limited 
changes, measured by performance measures. And 
they may contribute to larger impacts, measured 
by community-wide indicators. 



There is an important distinction between 
accountability for the performance of an 
Individual program, and for the status of 
larger populations. 



It is important that partnerships understand the 
distinction between accoimtability for an individual 
program, which can often show changes in a short 
period of time with limited resources and partners; 
and accountability for broader changes across larger 
populations, which usually require diverse coalitions 
working over longer periods of time. 

Babyland Family Services in Newark^ New Jersey 
works with many partners to sponsor a wide variety 
of child and family services^ including child care, home 
visiting, foster parent recruitment, medical services and 
parenting support. All are aimed at helping Newark's 
children learn and grow. Babyland can work with its 
partners to measure the impact of its combined activities 
on children and families in its target areas. 



2. To MANAGE THE WORK OF OTHERS 

(such as contractors or individual 

PARTNERSHIP MEMBERS). 

The partnership may decide to delegate or contract 
out some of its work to other individuals and 
organizations-usually specific activities such as 
direct services to families. In those cases, partner- 
ships can use performance measures to evaluate 
the achievements of these other individuals and 
organizations. Performance measures can tell the 
partnership how the activities affected the people 
served, and how this information should influence 
future program and funding decisions. For example, 
a partnership may contract with service providers 
to improve teaching practices and would work with 
them in taking on accountability for improvements 
among the teachers they serve. 

The Georgia Voluntary Prekindergarten Program 
provides funds for a variety of child care programs; its 
child care providers can measure the effectiveness of their 
work by examining performance measures among the 
children they serve. 

3. To CONTRIBUTE TO THE WORK OF OTHERS 

(such as peer organizations). 

Partnerships may be working with other organiza- 
tions and systems that are tracking progress in 
their own area, such as child and family health, 
educational performance or family wages. They 
may even decide to collaborate to take joint 
responsibility for community-wide indicators. 
Taking this step requires understanding how 
results-based decisionmaking works, the vocabu- 
lary and framework other organizations may use, 
and how to fit the different approaches together. 
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Most of the major results that society cares 
about cannot be achieved by one organiza- 
tion acting alone but require joint work 
across systems and communities. 

The Michigan Child Care Futures Project collaborates 
with the United Way of Saginaw County; United 
Way is using performance measures in its program 
evaluations, and is moving towards setting community- 



wide results and indicators. The two partners can 
collaborate to enhance each others understanding of 
using results, indicators and performance measures 
to improve service delivery. 

The figure below illustrates these three ways of 
using result-based decisionmaking. Note that the 
arrows go both ways. Effective and fair accounta- 
bility requires a two-way relationship, not one-way 
transfer of responsibility. 
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This guide is designed to help partnerships under- 
stand these three different ways of using results- 
based decisionmaking, and how they can use this 
process most effectively to improve the lives of 
children and families. The guide is intended to help 
partnerships recognize and manage the political 



issues around results-based decisionmaking, and 
understand enough about the technical considera- 
tions to be able to ask good questions and seek 
more information. 

The guide is organized into the following sections: 
(1) this introduction; (2) benefits of, and cautions 
about, using results-based decisionmaking; (3) the 
language of results-based decisionmaking; (4) the 
creation of “logic models,” which are frameworks 
for linking the partnership s desired results with 
activities to effect change; (5) guidelines for using 
results-based decisionmaking to improve perform- 
ance; (6) a phased approach for using results-based 
decisionmaking to improve the lives of children 
and families; and (7) a conclusion. The appendices 
also include resources for more detailed information 
on specific topics. 
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Benefits 




Considerations 



In the for-profit sector, measuring results — sales, revenue, 
profits — is a basic element of the work environment. Knowing 
who sold the most products, what items are most popular, and 
what are the revenue forecasts is essential for a profitable bottom 
line. Comparable information on the status of children is equally 



essential to efficient work with children and 
families. The difficulty has been that the obstacles 
to developing and using a fair and effective system 
loom large in the human service sector. Measuring 
the healthy development of children, and deciding 
who should be accountable for that result, is often 
more difficult than measuring and determining 
accountability for sales and profits of specific 
products. However, that’s not a reason to be daunted; 
it’s a reason to be cautious and to plan carefully. 

Reasons for Using Results- 
Based Decisionmaking 

■ Partnerships can use results-based 
decisionmaking to help generate interest 
in and commitment to improving the lives 
of children and families. 

Improving the status of young children often 
requires different sectors of the community to 
work together in new ways. It may require policy 
change to improve existing systems, decisions to 
redirecting existing resources and an infusion of 
new dollars. Implementing these decisions 
requires support from diverse community stake- 
holders, including public and private funders, state 
legislators, service providers, parents, voters and 
others. Using results-based decisionmaking can 
generate this support in two ways. 



First, having a well-managed public process to set 
priority results opens up the decisionmaking process 
to parents and other stakeholders, inviting them 
to be more invested in success. It can also reduce 
controversy and issues of turf among sometimes 
competing stakeholders, by building a process for 
them to collaborate on an agreed set of results. 

Second, taking responsibility for improved results, 
and then documenting progress, is a powerful tool 
to generate ongoing support. The combination of 
changing quantitative data, coupled with stories 
about children and families, can build a strong 
case for support among current and future partners. 



Data add substance to what could otherwise/ 
be dismissed as anecdotes, while stories" \ 
add a personal element to cold numbers . -■ 



on a page. 






Partnerships can use data to show potential sup- 
porters where they are going, why they are going 
in that direction, and the progress they are making. 
They can also show why supports for young chil- 
dren can be such a wise investment, both now and 
over time. 
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For example, a partnership is formed, and 
through public discussion and debate it 
decides that its joint aim will be to improve 
children’s readiness for kindergarten. The 
partnership uses a process to set its desired 
result and the strategies to affect that result 
that encourages joint responsibility for 
progress and that brings new players 
together. Making a commitment to measure 
and report on progress also generates 
confidence that the public will be able to 
see progress. 



■ Partnerships can use results-based deci- 
sionmaking to help them know where they 
are, where they want to go, and what 
progress is reasonable to expect. 

Partnerships can maximize their likelihood of 
success by learning how families and communities 
are faring before they decide what to do. Baseline 
data on the status of children and families can 
help a partnership decide where to go by knowing 
what issues are most pressing, how its community 
compares to other places or to research-based 
standards, and where its efforts can have the most 
impact. Setting joint results also requires parmers 
who may have different agendas to reach agreement 
on where they are trying to go together. Of course, 
data on the current status of children and families 
isn’t the only factor in setting the agenda — the 
partnership’s vision and goals, the political envi- 
ronment, individual members’ interests, and other 
considerations are also critical.' But objective data 
is a key ingredient to shaping effective strategies. 

Another key decision is deciding what progress is 
reasonable to expect. Knowing how a community 



compares to other places and agreed-upon stan- 
dards — as well as whether the numbers have been 
getting better or worse — is vital to making an 
informed decision about how far the partnership 
can expect to go. As Lisbeth Schorr has noted, 
using results helps to expose the myth that society 
can fix huge problems that have taken decades to 
develop, with tiny pots of money over relatively 
brief periods of time.^ All paitnerships need to 
decide what it will take to achieve their desired 
results, and what is reasonable to expect, given 
their resources and scope. 

When the partnership begins to study 
kindergarten readiness, it finds that many 
children aren't ready for kindergarten when 
they reach school age. Further study finds 1 
that a large number of these children have | 
been in child-care centers where the training 5 

for center staff is below recommended stan- ^ 

I 

dards, and research indicates a connection ^ 
between training and children’s improved 
development. The partnership decides to 
invest in a program to give child care teachers 
at particular centers more training, which j 
will, it hopes, cause them to use more ^ 
effective teaching techniques, which will | 
improve the quality of child care, which will 
improve the children’s development. The. | 
partnership realizes that improved develop- , 
ment takes more than training, and that 
changing the larger population will require ^ 
more resources than it has alone, but it 
decides to start there. This is the overall 
plan (or logic model) that describes where 
the partnership expects to go, how it will 
get there, how it will mark progress and ^ 
why it thinks this will work. 
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■ Partnerships can use results-based 
decisionmaking to help them track and 
improve performance. 

Partnerships are created to effect improvements, 
and they need to know and demonstrate if they 
are making progress. To do so, a partnership needs 
to know what performance measures it wants to 
affect, either through its own work or in collabora- 
tion with its contractors. Indicators then measure 
if the individual and joint initiatives across a 
community are adding up to progress towards the 




results. These measures help meet the need to show 
visible progress, to be accountable for performance, 
and to keep the partnership members (and funders) 
committed to the work. They can also be used as 
a management tool enabling the partnership to 
make corrections to keep improving its work. 



To carry the example forward, suppose that 
the partnership now designs and conducts a 
training program. It pays attention not only 
to how many teachers sign up and complete 
the course, but also whether it made any 
difference. One performance measure would 
be the degree to which providers in the pro- 
gram learned the material. Another would be 
whether the training had an impact on child 
development among the children served. 

So the partnership would need to measure 
changes in the behavior of teachers in the 
classroom, and changes in the children’s 
behavior. While the partnership’s program 
is likely to affect teacher learning, it will be 
harder to affect children’s development. But 
each set of performance measures — changes 
in teacher knowledge, changes in teacher 
behavior and changes in children’s develop- 
ment — would give information to the 
partnership and others about whether the 
partnership’s work was effective in changing 
teachers’ behavior and in helping children. 

It would also show what measures could 
not be impacted by a single program. This 
information in turns tells the partnership if 
something needs to be added or modified. 

So, the partnership then finds out that teach- 
ers who go through the program increased 
their understanding of child development, 
teaching techniques, effective discipline, etc. 
But they did not always use their techniques 
in the classroom, and children’s development 
did not change much. The $64,000 question 
is, what does the partnership do with this 
information? 
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■ Partnerships can use results-based 
decisionmaking to help them decide what 
to do next. 

Setting results and using indicator and pertormance 
measure data on children, families, and the early 
childhood system is essential to managing the 
partnership effectively. This may mean the part- 
nership manages its agenda or resources differently 
— taking on different activities, shifting money 
from one activity to another, asking new players to 
do old things, or inspiring old players to do new 
things. Faced with slumping sales figures, a com- 



The power of results and indicators is not 
just in collecting data-it’s in taking the 
much; harder step, of using those data to 
change what the partnership or its service 
providers are doing. 



pany would galvanize into action to figure out the 
cause. Is it a poor product? Changing demand? 
New competitor? Slump in the economy? Key 
salesperson out sick? All of these are possible, and 
each calls for a different solution. If the problem is 
new competition, and the company mistakenly fires 
its best salesperson, it’s made a critical mistake. To 
use results-based decisionmaking successfully, 
organizations need to find out; 

■ what is happening, 

■ why it is happening, and 

■ what could improve the partnership’s progress. 



The example continues. ..With the informa- 
tion described above, the partnership can 
celebrate what it has accomplished and 
increase its focus on what needs to be 
changed or added. It also realizes that its 
expectations about what it can accomplish 
need to be commensurate with its level of 
activity and amount of resources. The part- 
nership may realize that a good training 
course for child care providers can be 
expected to increase their knowledge. But 
it may not be enough to increase their 
performance and improve children’s overall 
experience. As it considers what to do next, 
it will want to note where its achievements 
on performance measures met expectations, . 
where they didn’t, why they didn’t, and what = 
should happen differently in the future. 

Based on these data, the partnership 
decides that it has a great teacher training 
course, but it needs to be supplemented 
with work to reduce child-teacher ratios, 
provide better teaching supplies, develop 
a mentoring component, and offer refresher 
courses. The story continues... 



A Few Considerations 
AND Cautions 

The move to results-based decisionmaking is a 
powerful shift — it can change everything from 
multi-year plans for billions of dollars, to an indi- 
iddual child care proidder’s ability to teach a child 
in the way she thinks will work best. But it is not 
without challenges. There are reasons why the 
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current systems have developed, and why they are 
so hard to change. A brief sketch of the challenges 
may help partnerships address them. 

■ Demand for quick, tangible change. There 
is a temptation to fund only those activities, and 
work with only those populations, that can generate 
quick and tangible changes. However, often the 
partnership needs to be able to support activities 
that do not generate changes so readily or easily. 
Partnerships (and their funders) should be realistic 
about what it takes to achieve desired results and 
how long it will take for change to be visible. 
Furthermore, they will need to decide how to 
address activities that are widely supported but 
virtually impossible to measure, such as services 
that are very informal, time-limited or anonymous 
(e.g., parent warmlines). (More on this later.) 

■ Attribution. Partnerships generally choose 
activities, performance measures and indicators, 
on the assumption that the activity will cause the 
desired change in the performance measures (and 
perhaps even in the indicators). Yet one of the 
most challenging aspects of shifting to results- 
based decisionmaking is being able to prove that a 
particular activity caused the targeted measure to 
change — that the change could be attributed to 
the activity. Indicators and performance measures 
may change for reasons unrelated to a particular 
activity, and claiming that an activity directly caused 
a measure to change requires a fairly sophisticated 
level of logic and data analysis. For this reason, 
partnerships need to be carefiil about the level of 
attribution they claim.^ Appendbc 3 gives more 
detail about measuring and analyzing data. 



■ Data availability. Its often difficult to find or 
measure data for many of the indicators and even 
performance measures partnerships care about, 
especially changes in behavior or knowledge in 
young children. Partnerships may need to find 
proxies, use creative evaluation methods, and 
know the limits of the data. 




■ Potential for unintended effects. 

Measuring changes in indicators or performance 
measures among specific subgroups of children 
and using those data to make decisions about 
services or placement for them can lead to harmful 
labeling or tracking of children. Accurately testing 
very young children is especially difficult (more on 
this later). For this reason, most experts do not 
recommend individual testing of very young chil- 
dren (except for diagnoses of specific disabilities or 
delays). Many experts also recommend creative 
testing methods, such as portfolios and asset-based 
methods, that may be less subject to misuse. 

■ Consequences. Using results, indicators and 
performance measures effectively is inherently 
controversial, since it involves assessing who or 
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what is productive, or not, and how to improve 
eflfectiveness. One of the toughest parts of results- 
based decisionmaking is creating a fair and effective 
relationship between a level of achievement on 
a performance measure and the consequences of 
that performance. Too often poor performance 
by a particular service provider is automatically 
associated with decreased funding, when other 
responses might be more effective at improving 
performance. Or the opposite problem occurs — 
funding continues to flow to ineffective service 
providers or services when it could be redirected 
to more productive uses. Partnerships need to be 
extremely careful to separate performance from 
consequences — to ensure that they know exactly 



why the program performed as it did before decid- 
ing what to do to improve. They also need to 
ensure there is a collaborative relationship between 
those who manage changes and those who work 
to produce those changes. 

■ Continued need for basic protections. 
Systems that use results-based decisionmaking still 
need to maintain basic health, safety and civil rights 
protections. As indicated in the first bullet above, 
people can use the demand for quick, tangible 
change as an excuse to serve only certain groups, or 
to skimp on activities that do not seem to contribute 
directlv to what is being measured. For example, 
ensuring that a child care center meets basic fire 
codes may not contribute visibly to improved out- 
comes — but those restrictions exist for good reasons. 
Therefore, there is a need to maintain certain basic 
requirements and boundaries to ensure that funda- 
mental protections are kept in place. 

These issues and concerns are not reasons to avoid 
results-based decisionmaking. But they do illustrate 
why partnerships need to be extremely careful to 
design decisionmaking systems that “do no harm.” 
The guidelines described later can help create a 
system that works for everyone. 



Notes 

1 See The Child Care Partnership Project/The Finance Project, A Guide to Succes^ Public-Private Partnerships for Child Care^ 
Washington, D.C.: U.S. DHHS/Child Care Bureau, 1999. 

^ Lisbeth Schorr, with Frank Farrow, David Hombeck and Sara Watson, The Case for Moving to Results-Based Accountability^ 
Washington, DC: Center for the Study of Social Policy, 1995. 

^ For a more detailed discussion of attribution, see Child Trends, Children and Welfare Reform: A Guide to Evaluating the Effects 
of State Welfare Policies on Children^ Washington, D.C.: author, no date. 
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The Language of Results 
Based Decisionmaking 



Concepts and Terms 



In order to measure results, partnerships must become familiar 
with the language of results-based decisionmaking/ Once the 
major concepts are clear, and the partnership has agreed on its 



own words for these concepts, partnerships will 
know that the members of their own group mean 
the same thing when they use the same words. 



And understanding their own terminology will 
help partnership members understand the differ- 
ent terms that other groups use-they will have 



their own “Rosetta Stone” that they can use to 
translate back and forth between the different 
vocabulary used by different groups. The figure 
and then the table below show the three major 
categories of terms used in this guide. (The terms 
are also defined in the glossary.) 
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Results: 

The “results'’ category is the general aim or out- 
come to which the partnership is trying to con- 
tribute. The framework starts here because any 
initiative, like any business venture, should start 
with what it wants to accomplish. This category 
reflects what communities and families ultimately 
want to happen for young children. Results gener- 
ally cant be measured directly but are the catch 
phrases that generate commitment and enthusi- 
asm for this work. A partnership working alone 
usually will not be able to achieve the result, but 
its work should contribute to it. 

Indicators: 

“Indicators” are the measurable elements of the 
desired result and show if the partnership is on the 
right path towards achieving its results. Results, 
indicators and the performance measures that 
reflect the effect of the work are “ends” rather 
than “meons”; they capture what a partnership 
wants to happen — changes in the people or 
systems the partnership wants to affect — not the 
measures of the partnership s activity to get there. 

Indicators reflect substantial changes — generally 
behavior or status — across an entire community. 
Influencing indicator data generally requires 
action by multiple partners rather than any single 
program or individual. 

Depending on the size of the community, the 
intractability of the problem, and the size of the 
investment, indicators can change in the shorter 
term (a few years) or take much longer to change. 

The concept of indicators can also be used to 
measure changes not only among children and 
families but also among systems or policies. 
Progress would be measured by “systems” indica- 



tors (such as enactment of a law) among a target 
population of decision makers (such as the legisla- 
ture). (The next chapter explores these two types 
of indicators in more detail.) 

Any given indicator can be expressed in differ- 
ent ways: 

^ Level of specificity: Indicators can be 
expressed in general terms (improved reading 
scores) or in terms of specific achievement levels 
within a specific population over a specific 
period of time (improve the percentage of 
children reading by the end of first grade by 
10 percent in 2 years within all public school 
children in this county). These more specific 
indicators can express what the partnership 
plans to achieve, or a particular standard 
against which the partnership will measure 
progress (e.g., progress towards a standard 
of 100% immunization). Partnerships may 
express their desired indicators in general 
terms initially, but they will need to provide 
the latter level of specificity when they move 
to determining the specific level of performance 
for which they are being held accountable, or 
for which they are holding others accountable. 

Indicators expressed in more specific terms 
called “benchmarl^lDr *Ynilesto^ 



m Rates or numbers: Indicators can be 
expressed in terms of numbers or rates; gener- 
ally both are necessary for a complete picture. 
Numbers give a sense of the scope of the 
problem or program, while rates are necessary to 
know if changes in the indicator are reflecting 
real differences in the status of the target pop- 
ulation. Knowing that the number of children 
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not reading by the end of first grade grew 
from 3,000 to 3,200 does not show whether 
this reflects an increase or decrease of the 
prevalence of the problem among the entire 
population. But knowing that the percentage 
of children not reading decreased from 10% 
of 30,000 children to 8% of 40,000 children 
not only reveals the scope of the problem, but 
shows that it is actually improving among the 
overall population. 

Performance measures: 

In a complete results-based decisionmaking system, 
it is important to consider how to allocate respon- 
sibility for change among the different partners. 
Each individual or program that is contributing 
to changing the community-wide indicator has a 
role to play. ''Performance measures'' reflect 
changes that one worker, stakeholder, program 
or organization can expect to affect. Performance 
measures are vehicles for individual programs to 
evaluate their own contributions to changes in 
indicators for the entire community. 

Performance measures can be divided into two types: 

■ ''Effect or product measures that reflect the 
impact of the work on children and families. 

These answer the question “Is anyone better 
off” because of the worL They can reflect 
changes in the knowledge, skills, attitudes, 
behavior, or status of the target population. 
Generally, those terms are in ascending order 
of difficulty in terms of both measuring and 
achieving them. That is, it s usually more 
powerful to say that people changed their 
behavior than to say that they learned some- 
thing, but learning something is still better 
than just reporting that they sat in a class for 



a certain number of hours. However, perform- 
ance measures by definition reflect changes 
that a single actor or organization can affect. 
Therefore, they tend to reflect smaller 
changes than indicators. 




However, they do share some similarities 
with indicators. Performance measures can 
reflect changes in people or systems; they can 
be expressed generally, or as a specific level 
of achievement; and as rates or numbers. 

It is important to point out that there can be 
some overlap between indicators and perform- 
ance measures. If a community is very small, 
or a program very big, the program-specific 
performance measure may be the same as the 
community-wide indicator. Also, very intensive 
programs, serving small groups of people, can 
sometimes affect the larger changes in behavior 
that are usually thought of as indicators. 

■ "Effort or process" measures that reflect 
the level or quality of activity. 

These reflect actions a partnership takes, the 
amount of services it provides, the products 
it produces, the money it raises or spends, 
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and the number of people it serves. Varieties 
include: 

• Materials that go into a program (funds, peo- 
ple, in-kind donations, volunteer hours, etc.). 

• The activity produced because of the pro- 
gram (number of newsletters or brochures 
published, classes held, etc.). 

• Measures of the quality of the program. 
Examples are error rates, lag time between 
referral and service, compliance with serv- 
ice delivery standards, accident rates, and 
customer satisfaction. For a child care center, 
one measure would be the student-teacher 
ratio. For a collaborative partnership, one 
measure would be the number and type of 
partners involved. 

Changing the definition of success to focus on 
results, indicators and the “effect” form of per- 
formance measures, rather than on the “effort” 
form of performance measures is fundamental to 
using results-based decisionmaking. However, 
measures of effort are stiU important. 

At the planning stage, partnerships need to con- 
sider what needs to go into a program in order to 



have a realistic chance of affecting what happens 
because of the program. Then at the implementa- 
tion stage, knowing how a service or activity is 
performing on effort or process measures is the 
first step in knowing if the service is on track to 
achieve the desired changes. For example, service 
providers need to have people coming to their 




programs if they hope to effect change. CoUabora- 
tives need to have the right partners at the table 
to hope to effect systemic change. Knowing how 
the programs are performing on these measures 
can also help the partnership evaluate why a serv- 
ice reached the desired achievement level on its 
performance measures or not. 
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Rgure 3: 

Definitions of terms 



Definitions 


Measurement 


Examples 


ENDS 


Result or Outcome: 

(Community’Wide) 

A condition of well-being for children, 
adults, families, or communities. 


The result is the vision or goal of the 
partnership. Results generally cannot 
be measured directly but rather are 
composites of various indicators. 


Children ready for kindergarten, 
children and youth succeeding 
in school, children in nurturing 
environments 


Indicator: 

(Community -mde) 

A measure, for \\^ch data are available, 
which helps quantify the achievement 
of the desired result for communin'- 
wide populations. 


Indicators can: 

• Change over a few years or several 
decades 

• Reflect changes in people or systems 

• Be expressed generally or as a specific 
level of achievement expected by a 
specific time 

• Be specified as rates or numbers 


Rates of children reading by the 
end of first grade, rates of children 
left alone after school before appro- 
priate age 


Performance measure: 

(Progrom’Specific) 

A measure, for \\^ch data are available, 
of the impact of agency or program 
service delivery; or the quantity or 
quality of agency or program service 
delivery. It quantifies the contributions 
of individual people or programs 
towards movement on the indicators. 


Performance measures reflect the 
contributions of individual stake- 
holders, workers, programs or 
organizations. Performance measures 
can measure either ‘^effect/product’* 
or ‘^effbrt/process”: 

Effect measures can be: 

• Changes in knowledge, skills, 
attitudes, behavior or status of 
the target population 

• Elxpressed generally or as a specific 
level of achievement by a specific 
time 

• Specified as rates or numbers. 


Rates of children completing home- 
work on time, rates of child care 
teachers using new skills. 


Effbrt/process measures can include: 

• “Raw materials” that go into a pro- 
gram — dollars, in-kind donations 

• Levels of activity — number of 
classes, products 

• Measures of the quality of a 
service — customer satisfaction, 
error rates. 


Number of students in classes, 
hours of staff training, student- 
teacher ratio. 


MEANS 
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One of the most difficult aspects of results-based 
decisionmaking is the distinction between meas- 
ures of effect and measures of effort. The table 
below describes some common areas of confusion. 
The elements in the first column are sometimes 



thought of as measures of effect. While they may 
be important measures and have their own uses, 
thev usually don t measure changes in the popula- 
tion. The third column shows what would be a 
better measure of effect. 



Rgure 4: 

Common pitfalls in defining measures 
of “effort” and “effect” 














Avoid: 


Because: 


Use Instead: 


Anecdotes about the impact ot | 
a service | 

I 


Stories can be very important to put a human face 
on othenvise cold numbers. But they are not, by 
themselves, sufficient to measure eifect or impact. 


Rates of changes of behavior 
among people served by a 
particular program 


Client satisfaction with a senice 


Parent satisfaction with child care is an extremely 
important measure and needs to be taken into 
account. However, it should not be the only measure 
of progress. Parents may be happy with mediocre 
child care, because they don't have other experi- 
ences to which they can compare it. Or a parent 
or teacher may be dissatisfied with a service (such 
as a challenging teacher training class) precisely 
because it is holding them to a high standard. 


Measures of client changes 
because of a service 


Descriptions of how hard 
people are working 


Improving children’s and families’ lives is hard, 
and people want to be acknowledged for how hard 
the}' in\ This acknowledgement should be an 
important part of any report. But measures of effect 
go beyond that level and talk about what happened 
because of the work, and what changes need to be 
made to reach higher levels of achievement. 


Measures of the changes that 
happen because of the work 
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Criteria for Selecting Results, 
Indicators and Performance 
Measures 

Choosing the right results, indicators and per- 
formance measures is key to an initiative that does 
the right things, measures the right things, and 
gets credit for the right things. This choice is 
every bit as much a political choice-reflecting 
what people care about-as a technical choice- 
reflecting what statistics are available and accurate. 
Here are three basic criteria to consider when the 
partnership is choosing its set of measures:^ 

■ Communication and political power: 
Partnerships should choose results, indicators and 
performance measures that the members and key 
stakeholders understand and care about, and that 
can persuade even skeptics to support the partner- 
ships work. While partnerships may need to 
measure some “smaller” performance measures 
that can change quickly (e.g., teacher knowledge 
among a specific population serv^ed), in order to 
mark progress and achieve some quick successes, 
it is the “larger” indicators (e.g., reading scores 
across the community) that carry the most politi- 
cal weight and will figure most heavily in policy 
decisions. The list of indicators should also be 
short enough to capture people’s attention and 
convey a clear message. 

■ Dafa power: Indicators and performance 
measures must be variables for which reliable and 
accurate data are currently and readily available. 
(More on this in appendix 3.) 




■ Proxy power: There should be a known rela- 
tionship among the performance measures, indica- 
tors and the results, either from research findings 
or best practices based on experience. The indica- 
tors should correspond to the status or events that 
the partnership wants to measure. 

Matching Expectations 
TO Resources 

Another key to success in using results-based 
decisionmaking is setting expectations about 
achievement that are commensurate with the part- 
nership s resources and timelines. No one really 
knows how much change in “performance meas- 
ures” is needed to add up to changes in communi- 
ty-wide indicators. Most partnerships will address 
only a part of the population, and only some of 
the factors that need to change in order to see 
larger effects in the community. The partnership, 
and the entities to which it is accountable, need to 
keep re-evaluating the match between what the 
partnership can effect, and what needs to happen 
to see community-wide results. 
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Using a logic model (more on this follows) enables 
partnerships to lay out exactly what they are 
doing, what impact they hope each actmty has 
on the people it serves, and how these programs 
are expected to “add up to” the desired changes in 
indicators. It can help prevent people from making 
claims far beyond what they hope to affect. Two 
points are key: 

■ Separate program-specific performance 
measures from community-wide indicators; 
expect progress only among the target 
population, systems, etc., that the partner- 
ship can serve with the funds and time that 
it has. One of the key mistakes often made in 
using results-based decisionmaking is expecting 
changes in the client population of one program 
to translate into changes in the status of larger 
populations. It is quite common for performance 
measures among clients of a particular program 
to show improvement, while indicators among 
the population at large continue to worsen. 
Partnerships should not expect to change the rate 
of school readiness for all children if they only 
have funds to serve 3% of the children who need 
help-or if they are only providing one element 
when the children need much more. An initiative 
that aims for even broader results, such as parents 
working, will need to include elements in addition 
to child care to reach its desired aim. However, the 



logic model shows how the package of strategies 
and activities is expected to contribute to the 
broader indicators. 

■ Use baselines to see what future perform- 
ance is reasonable to expect. A baseline 
charts indicators and performance measures over 
time. Partnerships need to review past data on a 
particular indicator or performance measure to 
have anv basis for estimating what a reasonable 
expectation of change would be. If the trends have 
been getting worse over time, just reducing the 
rate at which trends are worsening is a real victory 
— and should be treated as such. 



Partnerships may need to work towards 
their desired results in stages — and make^_ 
sure that they don’t make public claims that 
exceed what is reasonable to expect. . 



Partnerships shouldn’t be discouraged by limited 
resources. But they may need to work towards 
their desired results in stages-and make sure that 
they don’t make public claims that exceed what is 
reasonable to expect. For example, if a partnership 
is working toward school readiness, this years 
intervention may aim to improve children’s devel- 
opment in a limited area, with later expansion that 
will move closer towards the ultimate result. 



Notes 

Based on Mark Friedman, // Strategy Map for Results-Based Budgeting, Washington^ D.C.: The Finance Project, September 
1996; and Atelia Melaville. Guide to Selecting Results and Indicators^ Washington, D.C.: The Finance Project, May 1997. 

’ Adapted from Mark Friedman,// Strategy Map for Results-Based Budgeting, Washington, D.C.: The Finance Project, 
September 1996; and Atelia Melaville, A Guide to Selecting Results and Indicators^ Washington, D.C.: The Finance Project, 
May 1997. 
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Tying Results to 
The Logic Model 



Strategies: 



The next step in using results and indicators is to put them into 
a form that lets a partnership see how its strategies and activities 
are related to its desired results and indicators. One way is by 
using a “logic model.” A logic model is a framework that helps 
the partnership lay out its desired results, what affects those 
results, what it plans to do, and how it will measure progress. 



Partnerships can use the logic model in ways that 
correspond to the three “roles” described in the 
introduction: 

■ First, the logic model shows the relationship 
between the partnership s larger goal (its results), 
the activities it is pursuing to get there, and how it 
will measure progress along the way. This reflects 
the partnerships “theory of change” — its concept 
of what strategies and actions will cause desired 
changes in children s status and well-being.^ This 
helps the partnership MANAGE ITS OWN WORK, 
both initially and on an ongoing basis. It helps 
show if its expected changes are reasonable, given 
its resources and time. And it serves as a record of 
the partnership s assumptions, in case it does not 
have its desired impact. 

■ Second, partnerships can use a logic model to 
HELP MANAGE OTHERS’ WORK — to decide 
what work to “subcontract” to others. The work of 
these subcontractors should fit within the logic 
model; partnerships can even use the model as the 
basis for a request for proposals. 

■ Third, the logic model can show how the part- 
nership s specific activities can fit with the WORK 
OF OTHER ORGANIZATIONS TO ADDRESS ALL 



OF THE FACTORS NECESSARY TO ACHIEVE 
SOME RESULT. It can show in stark relief how 
difficult it is for any one entity to address all of the 
factors affecting the desired result. It is very possi- 
ble, and even common, for individual programs to 
do well, while indicators among the larger popula- 
tion do not improve. In that case, partnerships 
need to go back to the logic model to see what 
else it needs to do, or what other partners need to 
do, to move toward the desired result. 

There are many different logic models, just as there 
are many frameworks for learning about results- 
based decisionmaking. Below is one that can work 
for many public-private partnerships — partnerships 
may want to consider several approaches before 
choosing or modifying a particular model to fit 
their circumstances. Again, once the concepts are 
familiar, one would be able to apply the same 
understanding to any logic model. 

Bear in mind that any state or community effort 
to improve school readiness will inevitably be a 
complex undertaking. The logic model — like an 
architects scale model of a building — is not 
meant to be a detailed “blueprint” of what needs 
to happen. It is rather a place to lay out the major 
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Figure 5A: 

A Logic Model for Public-Private Child Care Partnerships 



Elements of the Model 




As Measured By 




Results 

i 

“The story” — 

CONDITIONS AND 
CAUSES 

i 

Strategies 

i 

Activities 


— 


Indicators 

Ultimate indicators 
Interim indicators 


/ Require \ 

/ community-wide\ 
1 EFFORT TO j 

\ AFFECT J 


Performance 

MEASURES 

Measures of effect 
Measures of effort 


/ Caw be AFFECTEfn 
BY THE 

\ PARTNERSHIP J 




This logic model builds on earlier work bv Mark Friedman but has been adapted to fit the particular characteristics of public- 
private child care partnerships. 
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Figure 56: 

A Logic Model for Public-Private Child Care Partnerships 



Elements of the Model 




As Measured By 




Results 

The partnership s overall vision for 
the well-being of children, families 
and the community. 

(e.g., children arriving at kindergarten 
ready for school), ^ 


— 


Indicators 

Ultimate indicators: 

Measure long-term elements of or 
progress towards the desired results. 
(e.g., children scoring well on kinder- 
garten assessments) 

Interim indicators: 

Measure shon-term elements of or 
progress towards the desired results. 
Reflects changes in the conditions 
or causes that affect achievement of 
the result. 

(e.g., percentage of certified teachers 
across a whole community, or percentage 
of children across a whole community 
scoring well on individual measures of 
school readiness) 


/ Require \ 

1 community-wide], 

1 EFFORT TO j 

\ AFFECT J 


“The story” — 

CONDITIONS AND CAUSES 

“The Story behind the numbers” — 
the conditions, causes, circumstances, 
assets, factors, issues, etc., that need to 
change to achieve the results. Based 
on research and knowledge about best 
practices. The partnership will work 
on some subset of these. 

(e.g., insufficient teacher training and 
skills to work effectively with young 
children) 


Strategies 

The broad approach or general action 
plan the partnership will pursue to 
change its chosen subset of conditions 
or causes, including the resources and 
partners required for success. 

(e.g, programs to improve opportunities 
for child care providers to learn and use 
new skills) 

i 


Performance measures 

Measures of effect: 

Measure changes in the target popula- 
tion served by a particular individual, 
program or organization. 

(e.g., child care providers in a particular 
program have greater knowledge about 
child development and demonstrate 
that knowledge by using more effective 
teaching techniques) 

Measures of effort: 

Measure activity — how much did 
the partnership do and how well did 
it do it. 

(e.g., the number of child care providers 
participating in professional development 
classes) 


1 Can be AFFECTEcn 

BY THE j 

\ PARTNERSHIP J 


Activities 

“What works” — the actions, programs, 
services, interventions, etc. that the 
partnership will use to implement 
the strategies. Includes resources and 
partners. 

(e.g, child care provider training work- 
shops, mentoring by master teachers) 
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strategies, to see how they fit together, and 
whether they can be expected to “add up” to the 
changes the partnership wants to see. 

Elements of the 
Logic Model 

The left column (results, conditions, strategies, 
activities) shows the basic elements of the logic 
model (and should be reflected in the partnership s 
action plan): 

■ Result: The logic model starts with the 
partnerships desired result — what is its vision, 
what does it ultimately want to happen or exist 
in its community? (Example: children arriving 
at kindergarten ready for school) 

■ Conditions and causes: The next step is to 
examine the “story” behind the results — what 
are conditions, causes, circumstances or factors 
that the partnership needs to affect to move 
towards the result. These reflect what causes the 
result to happen or prevents it from happening. 
These conditions should be based on research 
or at least specific experience about what works. 
(Example: insufficient child care provider train- 
ing and skills to work effectively with young 
children) 

■ Strategies: The next step is to examine the 
broad approaches or strategies that will affect 
those conditions or causes in order to reach the 
results. Both the strategies and the activities need 
to include the partners who will work with the 
partnership to implement the plan. (Example: 
improve opportunities for child care providers to 
learn and use new skills) 



■ ACTIVITIES: The final step is the activities, pro- 
grams, interventions, services, etc., that need to 
be implemented to carry out the strategies. 
(Example: increase child care provider training, 
provide mentoring by master teachers) 




Each activity the partnership pursues will probably 
change only one or a few of the conditions and 
causes that would need to change in order to reach 
the desired result. It may also affect only a subset 
of the total target population. So the partnership 
needs to think about how much activity it needs 
to bring about the desired improvements. 

Each element should also be based on research 
and/or experience that explain why a bad condition 
exists or a good condition does not exist (the con- 
ditions and causes) and what will fix it '(strategies 
and activities). 

Measurements of Progress 

The right column (ultimate indicators, interim 
indicators and performance measures) measures 
progress along each element of the logic model. 

■ Ultimate indicators reflect progress toward 
the result. They are usually a very few, important. 
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well-known measures that convey significant 
changes in status or behavior. They usually take 
a long time and require significant investment to 
change. The partnership s logic model reflects its 
expectation that its interim indicators and per- 
formance measures will contribute to the ultimate 
indicators. (Example: children scoring well on 
specific kindergarten readiness assessments) 

■ Interim indicators reflect the status of 
community-wide populations in the short term. 
They can be useful to document progress on the 
specific factors that the community feels will affect 
progress on the result. Successful performance 
can provide a sense of momentum to encourage 
the partnership and its partners to keep working 

One of the greatest difficulties in using 
results-based decisionmaking in the context 
of child care is developing and collecting data 
for indicators arid performance measures. 
While other fields such as health care and 
juvenile justice have a longer history of 
defining and collecting data for indicators, 
the early childhood field is at an earlier 
stage of development in identifying measures 
that are appropriate and feasible. 

towards the ultimate indicators. If the partner- 
ships performance measures are good, but the 
community-wide indicators are worsening, the 
partnership and its partners need to re-examine 
the plausibility that their individual and collective 
programs can affect the desired indicators. 
(Example: percentage of certified teachers across 
a whole community) 



■ Performance measures are of two types: 

• Measures of ''effect*' reflect the contribu- 
tions of individual workers, stakeholders, pro- 
grams or organizations towards progress on 
indicators. They allow the partnership to 
measure its own impacts and how they con- 
tribute to the larger changes. (Example: 
teachers in a particular program have greater 
knowledge about child development and 
demonstrate that knowledge by using more 
effective teaching techniques) 

• Measures of "effort" help the partnership 
evaluate how it is implementing its strategies 
and activities. These measures allow the part- 
nership to see if services are being delivered 
in ways that have the best chance of achieving 
the desired level of performance. They can 
show the partnership if it is on the right track 
before it can expect performance measures to 
change. (Example: the number of teachers par- 
ticipating in professional development classes) 

The following examples illustrate how child care 
partnerships could employ logic models to plan 
and track their work. 

Example One: Direct Services 

In this example, a public-private partnership has 
decided that it wants to improve school readiness 
of kindergarteners for children within a particular 
school district. One way to do this is to improve 
the quality and expand the availability of child care 
for these children. Again, it is important that the 
logic model reflect reasonable expectations for the 
resources and time available. The two logic model 
examples below show a few of the many factors, 
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Figure 6: 

A Logic Model for Direct Services to Children 
and Families to Improve School Readiness 



Elements of the Plan 




As measured by 


Result 




Ultimate indicators 


■ Improve school readiness of kindergarteners 




■ All children will enter kindergarten with 






readiness scores at or above the national norm 






Interim indicators 

■ More certified teachers across the entire 




community 


Conditions and causes 

■ Insufficient teacher training, which results in 
poorer quality care in certain centers 

■ Lack of available slots with high quality child 
care providers 

■ Etc. 




■ Children across the community show improve 
readiness on one or more dimensions 


i 






Strategies 






■ Teacher training 

■ Accreditation of child care centers and child 
care homes 

■ Etc. 

i 




Performance measures 

**Effecr 

■ Teachers in a particular program receive certifi- 
cadon of their increased knowledge and skills. 

■ More children from the targeted centers show 




behavioral, social and emotional readiness 


Activities 

■ In-service training class for child care aides 




Performance measures 


and teachers 




**Efforr 


■ Subsidize and provide technical assistance 




■ Number of child care providers participated in 


to centers and homes that work towards and 




the training 


become accredited. 




■ Number of children receiving care in newly 


■ Etc. 




accredited homes and centers 


(Note that this logic model describes only part of the factors, strategics, etc., involved in improving school readiness.) 
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strategies and activities that a partnership might 
want to pursue; a complete logic model would 
have additional information reflecting all of the 
community’s unique circumstances and preferences. 

Example Two: Policy Changes 

Improving results for children and families often 
requires not only changes in direct services to 
children and families, but also changes in the 
child and family system, including child care. 
Partnerships can track their progress in changing 
systems by using interim indicators of systems 
change. (Indicators of systems change are important 
because the partnership expects they will support 
ultimate indicators of improvements in child and 
family well-being.) 

The logic model can incorporate both services to 
children and families (measured by data on child 
and family well-being) and activities to change 
systems (measured by data on systems change). 



For example, in order to affect the ultimate indicator 
of “children ready for school,” a partnership may 
aim to change regulations or laws concerning 
teacher-child ratios, teacher certification require- 
ments, or reimbursement rates. In this case, changes 
that require multiple actors across a community — 
such as enacting new state laws — would be tracked 
by interim indicators. Changes that the partner- 
ship can affect — perhaps wages within a limited 
area as well as its activities to effect this change — 
would be tracked by performance measures. 

The figure below shows a logic model for systemic 
or policy changes to move towards the result of 
school readiness. 

Here the “conditions and causes” are systemic 
changes, such as changes in rules or processes that 
will themselves result in better services for children 
and families. Strategies and activities to change 
those conditions or causes involve work with systems 
rather than children and families directly. The 
Indicators and performance measures also reflect 
systemic change activities rather than direct services. 

Making progress on results will usually require a 
combination of direct services and policy changes. 
Developing the logic model helps partnerships 
be clear in their thinking and logic about what 
conditions and causes affect a desired result, and 
which strategies and activities are needed to 
address each element. 
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Figure 7: 

A Logic Model for Policy Changes to Improve 
School Readiness 



‘ Elements of the Plan 




' As measured by 


Result 




Ultimate indicators 


■ Improve school readiness of kindergarteners 




■ Children vrill enter kindergarten with readiness 


i 


•;x 


scores at or above the national norm 


Conditions and causes 




Interim indicators 


■ Insufficient child care facilities and therefore 




■ Creation of 500 new slots for licensed center 


child care slots, which means some children 




and home-based care 


are receiving care that does not prepare them 




■ New regulations enacted, or other voluntary 


for school 




compliance with lower ratios 


■ High teacher-child ratios and high teacher 




■ Policy changes that increase reimbursement rates 


turnover, both of which are linked with sub- 




from the government for publicly-sponsored 


standard care that does not prepare children 




child care, leading to higher pay; establishment 


sufficiently for kindergarten 




of a better career track for teachers, leading to 


■ Etc. 1 




higher satisfaction and lower turnover 


Strategies 

■ Expand child care slots 




Performance measures 


■ Improve conditions for child care teachers 




“Effecr 


■ Increase funding for new child care facilities 




■ Creation of a fund to support teacher wages 


■ Etc. 

t 




within a particular school 




■ Change in selected employers’ policies on 


Activities 




supporting high quality child care 


■ Elxpand available licensed center and home- 




Performance measures 


based child care for an additional 500 pre-school 




“Efforr 


children by expanding public reimbursement for 




■ Development of a written plan to create a 


child care, and creating a facilities fund to encour- 




facilities fund 


age development of more child care facilities 




■ Meetings with legislators on teacher-child ratios. 


■ Advocacy work to change regulations to reduce 




reimbursement rates, etc. 


teacher-child ratios 




■ Collaborative meetings on teacher turnover, recom- 


■ Establish a collaborative effort to increase 




mendations written and submitted to policymakers 


teacher satisfaction to reduce teacher turnover 




■ Number of employers and other partners in the 


■ Etc. 




collaborative 


(Note that this logic model describes only part of the factors, strategies, etc., involved in improving school readiness.) 



Notes 

* Carol Weiss, “Nothing As Practical As Good Theor)': Exploring Theoiy-Based Evaluation for Comprehensive Community 
Initiatives for Children and Families,” in James Connell, Anne Kubisch, Lisbeth Schorr and Carol Weiss, eds., 'Nev) Approaches to 
Evaluating Community Initiatives: Concepts^ Methods and Contexts (Washington, D.C.: The Aspen Institute, 1995). See also James 
Connell and Anne Kubisch, “Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: 
Progress, Prospects and Problems,” in Karen Fulbright- Anderson, Anne Kubisch and James Connell, eds., 'New Approaches to 
Evaluating Community Initiatives, Volume!: Theory, Measurement and Analysis (Washington, D.C.: The Aspen Institute, 1998). 
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Guidelines for 
Results-Based 



Using 

Decisionmaking 



Once the partnership understands the language and concepts 
around results-based decisionmaking, the next stage is deciding 
how to use them. The introduction to this guide describes several 
ways partnerships could use results-based decisionmaking. One 
is to manage its own work. Another is to manage the work of 



service providers and other entities the partnership 
funds. A third is to contribute to the work of other 
organizations responsible for results. The guide also 
describes three types of accountability relationships. 
One set of relationships is between the partnership 
and its creators or funders. ^\nother set of relation- 
ships is between the partnership and those service 
providers or other entities which the partnership 
funds and holds accountable for performance. A 
third is between the partnership and other coalitions 
working on their own child and family results. 
(See Figure 1.) 

These new roles and relationships represent a new 
way of operating for the partnership. As the “con- 
siderations and cautions” section above illustrates, 
this new way of operating is not without risks. It 
may be helpful for partnerships to consider guide- 
lines for establishing these new roles and relation- 
ships that will help avoid or minimize these risks. 



The guidelines will also help the partnership create 
a process for results-based decisionmaking that 
is both fair and effective — that contributes to 
improvements in indicators and performance 
measures, treats fairly those accountable for 
achievement and guards against unintended Ul 
effects towards children and families. 

In many cases, these guidelines are interrelated; 
implementing some without the others will weaken 
the overall impacts. For example, implementing 
the fifth guideline (moving resources from less 
effective strategies to more effective strategies) 
without the sixth one (maintaining basic health, 
safety and civil rights protections) could mean 
taking funds away from essential protections for 
children and families that may not have a demon- 
strable impact on indicators and performance 
measures. Each guideline is described in more 
detail following the table.^ 
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Guidelines for Using 
Results and Indicators 

1. Participants cleorly estoblish the terms of 
the performance contract and have the neces- 
sary knowledge and skills to participate 
in results-based decisionmaking. 

2. Performance measures, and the strategies 
to reach desired levels of achievement are 
negotiated, rather than dictated, 
between the partnership and those who 
are responsible for producing change. 

3. People and organizations are held account^ 
able for what they can reasonably expect 
to affect. 

4. Achievement on performance measures 
and indicators is tied to consequences, but 
the reasons for good or bad performance 
are analyzed to determine appropriate 
consequences. 

5. The consequences associated with performance 
move resources from ineffective Strategies 
to effective ones. At the same time, people 
and organizations that do not meet their 
goals are supported to move ahead, not 
fall further behind. 

6. There are'effe^ve safeguards to prevent 

inanities and.other ill effects, and to protect 
the basic health, safet}^ and civil rights of 
children and famili^. ^ J 



1. Participants clearly establish the 

TERMS OF THE PERFORMANCE CONTRACT, 

AND THEY HAVE THE NECESSARY KNOWLEDGE 
AND SKILLS TO PARTICIPATE IN RESULTS- 
BASED DECISIONMAKING. 

The “performance contract” is the formal agreement 
between those who are responsible for performance 
on indicators and/or performance measures, and 
those who hold them accountable.^ Assigning and 
assuming this responsibility can eventually involve 
high stakes — not only the lives and life chances 
of children and families but the status and resources 
of organizations and individuals responsible for 
working with them. Assigning and assuming 
accountability requires that participants clearly 
spell out the terms of the performance contract — 
who is responsible for what level of performance, 
over what period of time, with what resources, and 
Avith what consequences. Establishing and agreeing 
to these ground rules also requires new knowledge 
and skills. In order for partnerships, service providers 
and others to participate equitably, they need to 
understand indicators and performance measures, 
including how they should be chosen, what level 
of performance and what timeline is reasonable, 
and how consequences are assessed. 

2. Performance measures and the 

STRATEGIES TO REACH THE DESIRED LEVELS 
OF ACHIEVEMENT ARE NEGOTIATED, RATHER 
THAN DICTATED, BETWEEN THE PARTNERSHIP 
AND THOSE WHO ARE RESPONSIBLE FOR 
PRODUCING CHANGE. Many of those responsible 
for achieving a certain level of achievement on 
performance measures (or even indicators) are 
wary that funders will expect dramatic results 
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when in fact no one knows for certain how to 
reach those targets, or even what are reasonable 
expectations for performance. Or they are con- 
cerned that funders will expect them to affect 
community-wide indicators when they 
do not have the resources and authority necessary 
to do so. The partnership and the organizations 
responsible for performance should be able to 
negotiate which are the desired performance 
measures, the expected level of achievement, and 
the means to accomplish those targets (within 
logical parameters).^ 

3* People and organizations are held 

ACCOUNTABLE FOR WHAT THEY CAN REASON- 
ABLY EXPECT TO AFFECT. 

Part of the negotiation described above is to ensure 
a reasonable level of expectations for those who are 
to be held accountable. One of the most common 
objections to using indicators (and sometimes even 
performance measures) as a measure of success is, 
‘‘I cant be held responsible for (choose an indicator) 
since there are so many factors involved.” It is true 
that virtually all of the “ultimate” and “interim” 
indicators society cares about (child abuse and 
neglect, high school graduation, etc.) are the result 
of many factors, spread across the community. 

But there are a couple of ways to address this 
problem. One is to hold smaller programs responsible 
for more limited changes — a program to train child 
care workers could not be expected on its own to 
affect child development. But, depending on the 
duration and intensity of the intervention, it could 
be expected to affect performance measures such 
as teacher knowledge and behavior in the classroom. 

Second, this situation underlines the need for 
larger, collaborative structures to affect indicators 
(or at least the need for coordination among mul- 



tiple actors). Although one small project to train 
teachers cannot affect child development, a sustained, 
concerted effort involving private business leaders, 
the school system, the early childhood community 
and others, could hope to have such an impact. 
Indeed, one of the major benefits of results-based 
decisionmaking is that a focus on results can induce 
major programs and systems to work together to 
achieved shared goals — for example, child care. 
Head Start and maternal and child health agencies 
all have a role to play in improving child develop- 
ment. When child care partnerships work with 
these agencies jointly, agreeing to common goals, 
they can be jointly accountable for progress toward 
achieving joint results. 




Another way to look at this situation is that, in 
the private sector, one individual cannot control all 
of the conditions that go into whether his business 
is profitable and he makes money. A salesperson 
may work entirely on commission, and yet he does 
not have control over how his product is made, 
how quickly it is shipped, and whether his best 
customer just lost her job. In fact, most American 
workers’ salaries are tied to factors beyond their 
direct control. While there is not a perfect parallel 
between private sector production methods and 
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human services, there is enough experience with 
this dynamic in the private sector to expect that 
human services can begin to operate this way too. 

4. Achievement ON performance measures 

AND INDICATORS IS TIED TO CONSEQUENCES, 
BUT THE REASONS FOR GOOD OR BAD 
PERFORMANCE ARE ANALYZED TO DETERMINE 
APPROPRIATE CONSEQUENCES. 

Achievement on indicators and performance 
measures needs to be tied to some form of conse- 
quences, in order to complete the planning loop 
that uses information to constantly improve. These 
consequences can include incentive payments for 
good performance as well as penalties for poor 
performance. There are vastly different opinions on 
the best approaches to using consequences, with 
some states and systems advocating a longer time 
of assistance and peer pressure before moving to 
harder edged courses of action, such as de-funding, 
while others advocate moving more swiftly along 
that path. 

One of the greatest fears about using indicators and 
performance measures is that poor performance will 
automatically result in financial penalties, when 
that may not be either a fair response or one likely 
to produce better results. Instead, the partnership 
needs to insert an analytical step between learning 
about the achievement levels of a program, and 
administering consequences of that performance. 

A poor level of achievement may be the result 
of (1) the wrong strategy; (2) the right strategy, 
poorly implemented; (3) the right strategy, well- 
implemented but inadequately funded; (4) outside 
forces that no one anticipated; or even (5) faulty 
data that misrepresented the result. The partnership 
needs to examine the indicators and performance 
measures to determine the reason for poor perform- 



ance before deciding on the indicated course of 
action to improve. 

5. The consequences associated 

WITH PERFORMANCE INVOLVE MOVING 
RESOURCES FROM INEFFECTIVE STRATEGIES 
TO EFFECTIVE ONES. 

At the same time, people and organizations that 
do not meet their goals are supported to move 
ahead, not fall further behind. 

The ultimate reason for pursuing the difficult task 
of using results-based decisionmaking is to move 
scarce resources to more effective strategies. There 
are not enough resources to maintain funding for 
ineffective approaches while funding effective ones 
at the scale necessary to reach everyone in need. 
Indicators must be used to move resources towards 
those workers, organizations and strategies that 
prove themselves effective at improving the lives 
of children and families.^® 

At the same time, using results-based decision- • 
making should not mean abandoning children and 
families who do not do well. Those who do not 
meet their goals must be supported to move ahead, 
not fall further behind. While an effective system 
requires consequences that use achievement levels 
to constantly improve, a fair system acknowledges 
the vastly imperfect knowledge about the relation- 
ship benveen a chosen strategy or activity, and the 
effect on indicators and performance measures. 

It also takes into account the obstacles to good 
performance, including outside factors, populations 
that are more difficult to serve, and what the base- 
line data show. For example, improving on past 
performance can be difficult for all programs: those 
with good historical data, because they are already 
performing well, and those with worsening trend 
lines, because they face greater obstacles. 
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One of the greatest concerns about using results - 
based decisionmaking is that communities, part- 
nerships and programs that are the most likely to 
have difficulty achieving the desired indicators and 
performance measures, because of outside circum- 
stances, the difficulties they face, or other factors, 
are the very ones that need the most attention. The 
key to a successful results-based decisionmaking 
system is structuring consequences so that poor 
performance is translated into appropriate 
improvement strategies. 

This does not mean that an ineffective service 
provider is given a dozen more chances to fail its 
clients. If a service provider does not reach the 
expected level of achievement on performance 
measures s/he and the partnership examine the 
reasons together, and decide on a remedy, such as 
more training or mentoring. If a school does not 
improve, it is not penalized with a smaller budget 
but perhaps given leadership support, more technical 
assistance or more oversight. If a child is not ready 
for kindergarten, s/he is not automatically held 
back, but given more help. If a partnership shows 
improvement on performance measures, but the 
community-wide indicators are worsening, it and 
its funders need to examine the causes to deter- 
mine the best course of action. The focus is not 
on reducing funds for entities already desperate 



for resources, or otherwise instituting penalties 
that just aggravate the underlying causes for poor 
performance. 

6. There are effective safeguards to 

PREVENT INEQUITIES AND OTHER ILL EFFECTS, 
AND TO PROTECT THE BASIC HEALTH, SAFETY 
AND CIVIL RIGHTS OF CHILDREN AND FAMILIES. 

One of the most exciting aspects of using results- 
based decisionmaking is the implication that people 
will be given more responsibility for deciding how 
to accomplish the challenges in front of them. But 
flexibility also incurs risk If a school is rewarded 
for improving test scores, there is an enormous 
temptation to do so by not testing children who 
are unlikely to score well. If a particular group is 
likely to improve significandy, there is a tempta- 
tion to focus on that group and not another that 
will require more support. Therefore, there must 
be incentives to serve more difficult populations, 
and safeguards to ensure that indicators are not 
used as an excuse to exclude certain groups or 
otherwise misused. 

These guidelines, taken together, will go a long 
way towards ensuring that the partnerships’ use 
of results-based decisionmaking will be both fair 
and effective in improving the lives of children 
and families. 



Notes 

^ This section borrows from Beyond Lists: Moving to Results-Based Accountability, Washington, D.C.: Center for the Study of 
Social Policy, August 1996 (draft). 

® There is a large body of literature on performance contracting — for example, see citations by Bardach, Behn, Hatry and 
Walters in the bibliography. 

’ See also Mark Friedman, Trading Outcome Accountability for Fund Flexibility (Washington, D.C.: Center for the Study of 
Social Policy, no date). 

Sid Gardner, Beyond Collaboration to Results: Hard Choices in the Future of Services fir Children and Families. Fullerton, CA: 
Center for Collaboration for Children, January 1996. 



A Phased Approach to 
Results- Based Decisionmaking 

Once partnerships understand the basic definitions and concepts of 
moving to results-based decisionmaking, as well as the guidelines 
for using results-based decisionmaking fairly and effectively, they 
can consider how to move in this direction. There are several 
dimensions of results-based decisionmaking that vary in difficulty. 




level of controversy and potential to change services 
and systems. The ideas below are written specifical- 
ly for public-private partnerships. Larger entities, 
such as public agencies or whole systems, can go 
through similar dimensions. Partnerships may want 
to think about starting with less complicated and 
controversial dimensions (such as public informa- 
tion campaigns) and moving to more complicated 
phases (accountability). 

■ Public information and engagement: 

The most basic use of indicator and performance 
measure data is to know and share with the 
public the status of children and families — how 
they are faring now, and how that has changed 
over time. Partnerships can use these data to 
engage leaders in positions to make significant 
changes that will support improved results for 
children and families. The state of Vermont has 
used its public reports on the health and well- 
being of Vermonters to draw attention to the 
need for collaboration between the state and 
its citizens to “turn the curve” on the results 
the state cares about. The Annie E. Casey 
Foundations Kids Count programs in every state 
are examples of this use of indicator data (more 
information can be found at www.aecf.org). 



■ Planning: Another dimension would be to 
use data on indicators and some performance . 
measures to plan a partnership s agenda, including 
its results, how it will move towards those results, 
and how it will measure progress. The logic model 
is one way to plan the partnership s actions over 
the short and long term. 

■ Budgeting: Once partnerships have planned 
their agendas, their resources must be aligned to 
support this agenda. If a partnership uses some 
of its resources to contract for services delivered 



by other organizations, it should know what 
performance measures those contractors will 
address and how they fit into its plan to aflfect 
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indicators. Partnerships may also want to create 
‘performance contracts” that show who is 
accountable to whom, and for what level of 
performance with what population over what 
period of time.^^ 

■ Management and culture: Organizational 
culture, as well as management policies and 
practices, need to support movement towards 
desired indicators and performance measures. 
Both the partnership and its partners can use 
indicators and performance measures to manage 
their own work, building in incentives for good 
performance. But partnerships should bear in 
mind that shifting the culture of any organiza- 
tion is a huge undertaking, and moving to 



results-based accountabilitv requires significant 
cultural change. 

■ Accountability: Finally, a partnership can 
make itself accountable, to its funders, the public 
and others for moving towards its desired results. 
This requires not only measuring data to track 
changes in indicators and performance measures, 
but cycling back to the planning and budgeting 
phase to use that information to assess conse- 
quences, including redirecting resources, as 
necessarv; The ultimate purpose of results-based 
decisionmaking is to improve services and sup- 
ports in order to improve the lives of children. 
Indicators and performance measures that do not 
cause change are little more than list-making 
exercises; the purpose for going through all of 
this difficult work is to come out the other side 
with better supports and systems that improve 
the lives of children and families. This is by 
far the hardest step, but using the guidelines 
described earlier, and building a solid foundation 
before mo\ing to this stage, can help partnerships 
create a system that improves and expands child 
care while working fairly and efficiently with 
those who are accountable for performance. 



Notes 

See also Mark Friedman, Trading Outcome Accountability for Fund Flexibility (Washington, D.C.: Center for the Study of 
Social Policy, no date). 
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Conclusion 



Moving to results-based decisionmaking has the power to 
transform aspects of the way public-private partnerships support 

but for using their creativity and energy to create 
change. This is an exciting time, with exciting 
tools for progress, and child care partnerships are 
in an ideal position to use this potential to make 
long-lasting changes, in their own neighborhoods 
and across the nation. 



children and families. It can rebuild public faith 
in the ability to improve the lives of children and 
families. It can energize tired reformers who can 
now see progress due to their efforts. And it can 
catalyze needed changes among those who at 
last are rewarded not only for following the rules 
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Appendix i: Glossary 



Achievement levels: The expected or actual 
level of performance on an performance measures 
or an indicator by a specific population (e.g., 28% 
of children expected to be reading by second 
grade; at the end of the program 25% of children 
actually are reading by second grade). 

Activity: The specific actions — programs, 
interventions, services, etc. — that need to occur 
to implement the strategies. 

Condition: The causes, factors or circumstances 
that cause the result to happen or prevents it from 
happening, and that the partnership needs to 
affect to move towards the result. 

Indicator: A measure, for which data are avail- 
able, which helps quantify the achievement of the 
desired result for community-wide populations. 
Types of indicators: 

■ Child and family indicator: Measures 
that reflect changes in children and families 
(such as reduced rate of confirmed cases of 
child abuse) 

B System indicator: Measures that reflect 
changes in systems (such as enactment of a law) 

■ Ultimate indicator: Measures that reflect 
changes in results (usually change over several 
years and require community-wide efforts) 

■ Interim indicator: Measures that reflect 
changes in the conditions or causes that lead 
up to results (usually can change within shorter 
periods of time but still require community- 
wide efforts) 



Logic model: A framework that shows the 
relationship between the partnership s ultimate 
aim (its results) and the activities it is pursuing to 
get there, along with how it will measure progress 
along the way. The logic model also reflects the 
partnership’s ‘‘theory of change’’ — its theory 
about what affects the indicators and performance 
measures it cares about, and what works to improve 
them. It is a means for a partnership to plan its 
work and assess its progress, and it provides a 
vehicle for others to evaluate the logic of its 
actions as well. 

Performance contracts: Agreements 
between a body that is responsible for achieve- 
ment on indicators and performance measures, 
and another body to which it is accountable. 

The agreement should specify the desired level 
of achievement, among what population, in what 
period of time, and the process for determining the 
consequences of that performance (and, possibly, 
the consequences themselves). 

Performance measure: A measure, for which 
data are available, which helps quantify movement 
towards the indicator for specific target populations 
OR that measures the level of activity, efficiency, 
capacity or quality of a service or intervention. It 
can be affected by individual organizations or actors. 

ResU LT: A broad condition of well-being for 
children, adults, families, or communities, some- 
times also known as an outcome. 

RESULTS-BASED ACCOUNTABILITY: The 

phase of results-based decisionmaking in which 
organizations use results, indicators and perform- 
ance measures to constantly move closer to their 
desired result. 
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RESULTS-BASED budgeting: The process of 
aligning ones resources with the desired results, 
indicators and performance measures to support 
the strategies and activities necessary to achieve 
those results, indicators and performance measures. 

Results-based decisionmaking: The 

process of using results, indicators and performance 
measures to improve children and family well-being. 
Includes using these for public information and 
engagement; planning; budgeting; management; 
and accountability. 

Results-based planning: The process of 
establishing results, indicators and performance 
measures, and using them to plan strategies and 
activities to achieve those results, indicators and 
performance measures. 



Strategy: The broad approaches that the part- 
nership will use to affect the conditions or causes 
in order to reach the results. 

Target population: The people whose skills, 
knowledge, behavior, etc., the activities are meant 
to influence, and among whom the indicators will 
be measured (such as all children within a specific 
school district, for child and family indicators; and 
legislators, for a system indicator). 

Theory of change: The partnerships assump- 
tions and decisions about the changes it is seeking, 
what influences that change, and what must happen 
to bring about that change. 
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Appendix 3: Issues in Selecting and 
Measuring Data on Indicators 




This appendix provides a bit more detail on the 
technical issues associated with selecting and 
measuring data on indicators. 

An essential part of using results and indicators is 
the ability to collect and analyze data. The part- 
nership may use experts to help with the technical 
issues around data collection — often partnerships 
can find help with evaluation and data analysis 
from faculty and/or graduate students at local col- 
leges and universities. While partnership members 
may not need an in-depth knowledge of statistical 
techniques, they do need some basic familiarity 
with data collection and analysis. Below are some 
of the key concepts and terminology used in 
measuring indicators: 

■ Fitting the indicators and the data 

COLLECTION PROCESS TO THE STRATEGY 
AND ACTIVITY: Comprehensive, long-term 
strategies and activities call for a more extensive 
and formal data collection process. By the same 
token, smaller, more informal services often 
cannot afford and should not be compelled to 
have the same level of data collection. A service 
provider can use indicators to measure the 
impact of a S2,000 grant to provide books to 
child care centers and a $1 million initiative to 
reduce teacher-child ratios and raise teacher pay. 
But in the first example, the service provider 
might use interviews with child care teachers to 
estimate if kids are reading more, while in the 
second the provider would use more formal 
techniques to see if children were learning and 
showing development gains. And indicators for 
some interventions simply cannot be measured 
in any meaningful way — in this case, the part- 
nership might look at the package of interven- 
tions to see if it makes sense that the whole 
package is needed to effect change. 



■ Choosing the most accurate indicators: 

Indicators need to be reliable and valid. 

“Reliable” means that they measure the same 
situation the same way each time — so the same 
test administered to two different people vdth 
the same situation would show the same results. 
“Valid” means that they measure what they are 
intended to measure. Indicators also need to 
reflect data that can be measured accurately over 
time, so that an evaluation today and next year 
provide numbers that can be compared against 
each other. Finally, they need to be readily inter- 
pretable, so that one would know if a change in 
numbers means the situation is getting better or 
worse (often one might need to look at several 
indicators together to understand whether the 
situation is improving or not). 

■ Knowing if a change has occurred: 

PRE- AND POST-TESTS: The clearest way to 
know if a person has changed during an inter- 
vention is to collect data on the same indicator 
before and after the intervention. So most data 
collection methods will try to gather the same 
data (use the same questionnaire) before, or at 
the beginning of the service (the pre-test) and 
after, or at the end of the service (post-test). The 
difference in the subjects’ knowledge, behavior, etc., 
as shown by the change between the two meas- 
urements, is the performance on the indicator. 



■ Knowing if the change can be attrib- 
uted TO A PARTICULAR INTERVENTION: 
CONTROL GROUPS: As described above, one of 
the trickiest aspects of using indicators is knowing 
that a particular performance on an indicator was 
the result of a particular intervention. Indicators 
can change for a variety of reasons, and even 
if two data elements are correlated (or change 
together), the first does not necessarily cause the 
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other. They could move together for independent 
reasons, or instead of the first variable causing the 
second to change, the reverse could be true. For 
example, if children of a certain teacher perform 
better on tests, it could be because of the teachers 
superior training, or because her classroom has 
more windows, or because she is assigned students 
who already tend to perform better on tests. 

The best way to establish causality or attribution 
is to have a “treatment” group, that gets the 
service, and a “control” group that is similar in 
every way except that it doesn’t get the service. 
Then one can be fairly sure that if the treatment 
group behaves differently, the difference is due 
to the intervention. 

But this standard is extremely difficult to achieve 
in social sciences. For example, it s often seen 
as unethical to withhold the service, or families 
often find ways of getting some service even if 
they are put in the “control” group. 

So evaluators often have to find other ways to try 
and determine if the service made a difference. 
Sometimes they try to look at the same child 
earlier in life and then compare that with informa- 
tion later in life, after the service (“longitudinal” 
studies). They look for similar groups of children 
in other geographic areas who aren’t getting the 
service (“natural control groups”). They sometimes 
use anecdotal information from participants to 
see if there’s evidence that it was indeed that 
service that made the difference. 

Another option is to examine the quality of the 
interventions theory of change and logic model 
to see if it seems reasonable that this intervention 
would be effective. While this method does not 
actually use indicator data, it does use the prin- 
ciple of results-based decisionmaking to estimate 



the likelihood that the intervention is having an 
impact. This evaluation strategy can be used in 
systemic or community-wide initiatives or with 
very informal, time -limited or anonymous serv- 
ices, where indicator- based evaluation methods 
would not be feasible. 

■ Choosing a method and instrument to 

MEASURE THE INDICATOR: There are many 
instruments designed to examine every aspect of 
a child or parent’s functioning. But choosing and 
administering one is a complex task. There is 
controversy over the costs and benefits of higher- 
cost techniques (testing children individually, 

‘ using trained evaluators with detailed assess- 
ments) versus lower-cost techniques (using more 
generic assessments with evaluators who know 
the children but who may not have had much 
training in the assessment, such as kindergarten 
teachers). There are issues about how intrusive 
the instrument is to the child or family. And there 
are controversies over exacdy how accurate each 
is at measuring the particular aspect of behavior 
or knowledge that partnerships may want. 

■ Addressing particular concerns 

ABOUT ASSESSING YOUNG CHILDREN: 

There are special concerns about assessing young 
children. Young children grow at different rates, 
and often do not easily demonstrate what they 
know and can do. Many child development experts 
believe that testing of young children has been 
used to label or stigmatize them inappropriately, 
and that different results are more a reflection 
of inaccurate tests or different developmental 
stages than true deficits needing attention. 
Partnerships need to be especially careful when 
measuring the results of their work among 
young children to ensure that the results are 
accurate, and are not misused. 



Notes 



‘2 For more information about assessing young children, see Shepard, Lorrie. Sharon Lvnn Kagan, and Emily Wurtz, eds., 
Principles and Recommendations for Early Childhood Assessments. Washington, D.C.: National Education Goals Panel, 1998. 
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Appendix 4: Selected Instruments 
to Measure Collaboration 

(Written bv William Philliber ot Philliber Research Associates) 



Instrument 


Dimensions 


Description 


Comprehensive School Health 
Program Infrastructure 
Development 
(Academy for Educational 
Development, 1255 23rd St., NW, 
Washington, D.C. 20037) 


Coalition existence 
Governance 
Information 5c training 
Needs assessment 
Long-range planning 


10 items are rated on a four point scale 


Interagency Collaboration 
Checklist 

(Center for Applied Research and 
Educational Improvement, 

265-2 Peik Hall, 159 PiHsbur>' Dr. SE, 
Minneapolis, MN 55455) 


Interagency coordination 
Interagency domain conflict 
Interagency satisfaction 


30 items are rated on Likert type scales 


Organization Dimensions Scale 
(Center for Applied Research and 
Educational Improvement, 

265-2 Peik Hall, 159 Pillsbury Dr. SE, 
Minneapolis, MN 55455) 


Satisfaction wdth: 
Goals 

Leadership style 
Coordination 
Decision making 
External conditions 


28 items are rated 


Tools for Assessing 
Collaboratives" and Community^ 
based Organizations' Capacity to 
improve Outcomes for Children 
and Families: Local Collaborative 
Assessment of Capacity (Center 
for Collaboration for Children, 
California State University, Fullerton, 
CA 92634) 


Governance 5c Accountability 

Outcomes 

Financing 

Non-financial resources 
Ownership 

Staff 5c Leadership Development 
Program strategies 
Policy agenda development 
Interorganizational coherence 
Addressing equity issues 


57 items are rated on a nine point 
scale 
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Instrument 


Dimensions 


Description 


juvenile Crime Prevention 
Project Collaboration Survey 
(Philliber Research Associates, 28 
Main Street, Accord, NY 12404) 


Reason for participation 
Meetings 
Communication 
Interagency staff relations 
Ownership 

Knowledge of other agencies 

Politics 

Conflict 

Direction of action 

Barriers 

Assets 


Individuals rate 47 items on a four 
point scale to measure 11 dimensions 


Survey for Directors of 
Collaborative Services Programs 
(National Center on Education in 
the Inner Cities, Temple University, 
Philadelphia, PA 19122 
(215) 204-3001) 


Interagency relations 
Communication 
Financing 8c Resources 


35 items are rated on a five point scale 
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Appendix 5: Selected Instruments 
to Measure Quality of Child Care 



Instrument 


Outcomes Measured 


Description 


Caring for Our Children: National 
Health & Safety Performance 
Standards for Out of Home 
Child Care Programs 
(American Public Health Assn., 

1015 15th Street, NW, 

Washington, D.C. 20005) 


Staffing 

Program 

Nutrition 6c Food Service 

Facilities, Supplies, Equipment 6c 

Transportation 

Infectious Diseases 

Children with Special Needs 

Administration 


Detailed description of standards 
and rationale with guidelines for 
measurement 


Early Childhood Environment 
Rating Scale 
(Teachers College Press, 

1234 Amsterdam Ave., 

New York, NY 10027) 


Range from safety of equipment to 
qualitv of teaching 


Assesses the day-to-day quality of 
care provided to children 


Family Day Care 
Rating Scale 
(Teachers College Press, 
1234 Amsterdam Ave., 
New York, NY 10027) 


Space 6c Furnishings 
Basic Care 

Language 6c Reasoning 
Learning Activities 
Social Development 
Adult Needs 

Provisions for Exceptional Children 


40 items rated on scale from 1 to 7 
Interrater reliability above .90 
Consistent with stand 2 irds of Child 
Development Associate Family Day 
Care credential 


High/Scope Program Quality 
Assessment (High/Scope Press, 600 
N River St.» Ypsilanti, MI 48198) 


Learning Environment 
Daily Routine 
Adult-Child Interaction 
Curriculum Planning 6c Assessment 
Parent Involvement 6c Family Ser\ices 
Staff Qualifications 6c Staff 
Development 
Program Management 


Assesses center-based pre-school and 
child care programs 
72 items rated on scale from 1 to 5 
Interrater reliability for exact agreement 
approximately .80; close agreement 
approximately .97 


tnfant/Toddler Environment 
Rating Scale 
(Teachers College Press, 

1234 Amsterdam Ave., 

New York, NY 10027) 


Furnishing 6c Display 
Personal Care Routines 
Listening 6c Talking 
Learning Activities 
Interaction 
Program Structure 
Adult Needs 


Assesses center-based child C 2 ire for 
children up to 30 months 
35 items rated on scale from 1 to 7 
Consistent with Criteria for Quality 
Early Childhood Programs stated 
by the National Academy of Early 
Childhood Programs 
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Instrument 


Outcomes Measured 


Description 


School-Age Care Environment 
Rating Scale 
(Teachers College Press, 

1234 Amsterdam Ave., 

New York, NY 10027) 


Space 6c Furnishings 
Health 6c Safety 
Activities 
Interactions 
Program Structure 
Staff Development 
Special Needs 


Assesses center-based child care for 
elementary school aged children 
49 items rated on scale from 1 to 7 
Interrater reliability for the total scale 
measured at .83 


Standards of Excellence for Child 
Day Care Services 
(Child Welfare League of America, 
440 First St., NW, Suite 310, 
Washington, DC 20001) 


Child Day Care Program 

Facilities 

Staffing 

Administration 

Suppon Services 


Describes the components of a quality 
program in each area. Can be used as 
a check-list to assess quality. 
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Appendix 6: Selected Instruments 
to Measure Child Development 



Instrument 


Outcomes Measured 


Description 


Bayley Scales of Infant 
Development 
(Psychological Corporation, 
555 Academic Court, 

San Antonio, TX 78204) 


Mental development 
Motor skills 


For ages 1 to 42 months 
Standard scores by age 


Bracken Basic 
Concept Scale 
(Psychological Corporation, 
555 Academic Court, 

San Antonio, TX 78204) 


Concept acquisition 
Receptive language 


For ages 2.6 to 8.0 
Standard scores by age 


Core Observation Record 
(High/Scope Press, 

600 North River Street, 
YpsUanti, MI 48198) 


Initiative 
Social relations 
Creative representation 
Music 8c movement 
Language 8c literacy 
Logic 8c mathematics 


Based on observation of child’s 
behavior during normal program 
activities 

For ages 2.5 to 6.0 years 


Developmental Indicators for the 
Assessment of Learning (DIAL) 
(American Guidance Service, 

PO Box 99, Circle Pines, MN 55014) 


Motor skills 
Concept skills 
Language skills 

Social/emotional Development 


Designed to assess large groups quickly 

8c efficiently 

For ages 2.0 to 5.11 


Differential Abilities Scales 
(Psychological Corporation, 

555 Academic Court, 

San Antonio, TX 78204) 


Verbal Ability 
Nonverbal Ability 
Diagnostic Ability 


For ages 2.6 to 6.0 
Standard scores by age 


FirstSTEP 

(Psychological Corporation, 
555 Academic Court, 

San Antonio, TX 78204) 


Cognition 

Communication 

Motor 


For ages 2.9 to 6.2 
Scaled scores by age 
Able to do in 15 minutes 


Kaufman Assessment Battery 
for Children 

(American Guidance Service, 

PO Box 99, Circle Pines, MN 55014) 


Sequential processing 
Simultaneous processing 
Achievement scale 


For ages 2.5 to 12.5 


Kaufman Survey of Early 
Academic and Language Skills 
(Western Psychological Services, 
12031 Wilshire Blvd., 

Los Angeles, CA 90025 


Expressive skills 
Receptive skills 
Number skills 
Letter 8c word skills 


Normed for 3 to 6 year olds 
Administered in 15-25 minutes 
Used to assess school readiness 
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Instrument 


Outcomes Measured 


Description y'/ 


Kindergarten Readiness Test 
(Slosson Educational Publications, 
PO Box 280, 

East Aurora, NY 14052) 


Understanding, awareness, 5c interaction 

with environment 

Judgment & reasoning in problem 

solving 

Numerical awareness 

Visual fine-motor coordination 

Auditory attention span & concentration 


For 4 to 6 year old children 
Primary purpose to assess child’s 
readiness for school 


McCarthy Scales of Children's 
Abilities 

(Psychological Corporation, 

555 Academic Court, 

San Antonio, TX 78204) 


Verbal 

Perceptual-Performance 

Quantitative 

Cognitive 

Memory 

Motor 


For ages 2.5 to 8.5 
Standard scores by age 


Miller Assessment 
for Preschoolers 
(Psychological Corporation, 
555 Academic Court, 

San Antonio, TX 78204) 


Motor 

Coordination 
Verbal 
Nonverbal 
Complex tasks 


For ages 2.9 to 5.8 
Percentile ranks 


Minnesota Child Development 
Inventory (American Guidance 
Service, 

PO Box 99, Circle Pines, MN 55014) 


Expressive language 

Comprehension/conceptual 

Personal/social 

General development 

Gross motor 

Fine motor 

Situational comprehension 
Self-help 


Parents complete inventory after 
observing child 
For ages 0.5 to 6.0 


Mullen Scales of 
Early Learning 

(Western Psychological Services, 
1203lWdshire Blvd., 

Los Angeles, CA 90025 


Gross motor skills 
Fine motor skills 
Visual reception 
Elxpressive language 
Receptive language 


For ages birth to 5 years 8 months 


Peabody Picture Vocabulary Test 
(American Guidance Service, 

PO Box 99, Circle Pines, MN 55014) 


Receptive language 


For ages 2.5 to 18.0 
Standard scores by age 


Preschool Inventory (Educational 
Testing Service, 

1947 Center St., Berkeley, CA 94704) 


Personal/social 
Associative vocabulary 
Concept activation-numerical 
Concept activation-sensory 


For ages 3. to 6 
15 minutes to administer 
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Appendix 7: Selected Instruments 
to Measure Family Functioning 





Instrument 


Outcomes Measured 


Description 


Family Assessment Form 
(Children’s Bureau of Southern 
California, 50 South Anaheim Blvd.. 
Suite 241, Anaheim, CA 92805) 


Family’s environment 
Physical 
Financial 
Support 
Caregiver(s) 

History 

Personal characteristics 
Child-rearing 
Family interaction 
Parent to child 
Child to parent 
Parent to parent 


Staff rate family on 61 items using a 
scale from one to five 


Family Assessment Measure 
(Addiction Research Fdn., 

33 Russell St., Toronto M5S 2Sl) 


Task Accomplishment 
Role Performance 
Communication 
Affective Expression 
Involvement 
Control 

Values 6c Norms 


Assesses the family from 3 perspectives: 

1. General (health of the family 
from a systems perspectives) 

2. Dyadic (relationships among 
specifrc pairs) 

3. Self-rating (own functioning in 
the family) 

Self-administered questionnaires 


Family Environment Scale 
(Consulting Psychologists Press, 
577 College Ave., 

Palo Alto, CA 94306) 


Relationships 

Cohesion 

Expressiveness 

Conflict 

Personal Growth 
Independence 
Achieve Orientation 
Intellectual/Cultural Orientation 
Active/Recreational Orientation 
Moral/Religious 
System Maintenance 
Organization Control 


90 item self administered questionnaire 
Normed for normal families 
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Appendix 8: Selected Instruments 
to Measure Parenting Skills 
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Instrument 


Outcomes Measured 


Description 


Adult'Adolescent 
Parenting Inventory 
(Family Development Resources, 
Eau Claire, WI 54703) 


Age appropriate Elxpectations 
Empathy toward child 
Belief in physical Punishment 
Role expectations 


For parents of preschoolers 
Normed scores 

Self-admiiiistered 30 item quesdonnaiie 


Nursing Child Assessment 
Teaching Scales 

(NCAST, University of Washington, 
Seattle, WA 98195) 


Sensitivity to cues 
Response to child distress 
Sodal-emotional growth Fostering 
Cognitive growth fostering 
Clarity of cues 
Responsiveness to parent 


73 items assess teaching interaction 
between caregiver and child 
For ages birth to 3.0 years 


Parent Awareness 
Skills Survey 

(Western Psychological Services, 
12031 Wilshire Blvd., 

Los Angeles, CA 90025) 


Assesses awareness of: 

! Critical issues 
Adequate solutions 
Need to communicate 
Importance of child’s history 
Need to pay attention to child’s response 


Assjesses how well a parent responds 
to typical child-care situations 


Parent Behavior Checklist 
(Clinical Psychological Publishing Co., 
4 Conant Square, 

Brandon, VT 05733) 


Expectations 

Discipline 

Nurturing 


For parents of children 1 through 
4 years 

Normed scores by age of child 
Self-administered 100 item 
quesdonnaire 


Parent-Child Relationship 
Inventory 

(Western Psychological Services, 
12031 Wilshire Blvd., 

Los Angeles, CA 90025) 


Parental support 

Satisfaction with parenting 

Involvement 

Communication 

Limit setting 

Autonomy 

Role orientation 


For parents of children 3 through 15 
Requires only 4th grade reading level 
15 minutes to complete 


Parent Satisfaction Scale 
(Psychological Corporation, 
555 Academic Court, 

San Antonio, TX 78204) 


Satisfaction with: 

Spouse’s parenting performance 
Parent-child relationship 
Parenting performance 


Self-administeied 46 item quesdonnaiie 
Appropriate for parents of any 
age child 

Normed scores and percentiles 
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